Applying Familiar Non-Parametric Tests to Evaluation of Neural Network Inputs

Feed-forward multi-layer neural network models, such as back-propagation networks with one or more hidden layers, present special challenges when it is necessary to evaluate and rank the contribution of each input variable. A number of methods have been proposed, including the multivariate Wald test and sensitivity analysis. Neither of these methods produces a complete and satisfactory answer to the question: is this variable significant? This is because these classes of neural networks produce models which are nonparametric and for which the optimal solution may not be unique. In addition, the contribution of any variable may depend on other variables due to the non-linear shape of the typical s-curve transfer function. Thus, a methodology for applying non-parametric tests must include both the test itself, and some form of monte carlo method of mixing and matching input elements to see if the tested variable is both independent and significant. The SAS language PROC NPAR1WAY facilitates non-parametric testing. The interpretation of the results produces relative rankings of each variable and significance levels. The methodology, results, and caveats/limitations are described in detail. INTRODUCTION Data are expensive to collect and maintain, and it is possible that the costs of collecting and maintaining a particular data element outweigh the benefits of having it available. Although scanning devices are available, much valuable data is still obtained through the labor intensive devices of observation, measurement, or surveys. In parametric-based multivariate analysis, e. g., regression analysis, widely accepted methods exist to determine the direct contribution of any given input to the result. The parameter estimates hold the key to the relationship between independent variables (inputs) and the dependent variable (outcome). Multi-layered feedforward neural networks, i. e., those with at least one hidden layer, present special challenges, however. First of all, there is not a unique one-to-one association of any input to any parameter; the relationship between inputs and outputs are represented by multiple connections. These connections feed weighed values to a number of hidden layer units, and then another set of connections from the hidden units to the output units represent a combined contribution of the inputs to the outputs as defined by the hidden layer activation. Second, each layer may have an activation or transfer function that is non-linear, such as a hyperbolic tangent or sigmoid function. Third, if all connections are allowed to vary with complete freedom in the training or development process, any optimal solution is not necessarily unique. White provides a detailed overview of neural networks focusing on their statistical properties in [10]. Certain types of multivariate analysis applications require determination of significance of individual inputs. For example, in determining the efficacy of a specific dosage of a drug, the dosage, patient age, and patient weight may all have independent or interdependent effects on the outcome. In other applications, it may be necessary only to know the relative contribution of an input to the outcome. For example, in a marketing response model, if "number of years at same address" is the least significant contributor, the decision can be made regarding the cost-benefit of maintaining that piece of data. Here, it is proposed that non-parametric tests of significance, including some of those based on the Empirical Distribution Function (EDF), provide a methodology for assessing the statistical significance of individual inputs in a feed-forward multi-layered neural network. A brief tutorial on EDF's and their related tests is included, and a neural network example with results is discussed in detail. Madansky [7] devotes a substantial amount of discussion in his book to non-parametric procedures and tests from a very practical viewpoint, and is recommended for supplementary reading. MEASURES OF INFLUENCE AND SIGNIFICANCE Two previously proposed methods of evaluating the significance of neural network inputs are the Wald statistic and sensitivity analysis. The Wald test is parametric, meaning that the test actually tests the hypothesis that a parameter or subset of parameters is zero, therefore inferring that the input that is associated with that parameter is insignificant. Conceptually, the Wald statistic is derived by taking the difference between the total of squared deviations for both the constrained (some parameters set to 0) and unconstrained models, multiplying that by the total number of observations and then dividing by the total of squared deviations for the unconstrained model. Amemiya [1] discusses a number of forms of the Wald statistic for both linear and non-linear models. Kuan and White [10] discuss the limitations of using the Wald test for multi-layered neural networks, and Golden [2] describes a procedure for applying the Wald test to any processing node of a multi-layered neural network. Essentially, the Wald test cannot be applied "system-wide", but only to a subset of connection weights connecting one layer to the next layer (peer-level). Sensitivity analysis measures the percentage change in output caused by a percentage change in input. NeuralWare [8] includes sensitivity testing as part of its software, and Hashem [3] proposes a set of equations for conducting sensitivity analysis on neural networks. Hashem's methodology is more generalized than that offered by NeuralWare, but neither produces a system-wide evaluation of the contribution of any given input to the output. In addition, there is no test statistic produced for the construction of inferences. Therefore, neither the Wald statistic nor sensitivity analysis will provide a statistical inference about the significance of an input in a multi-layered neural network that can be compared to more traditional modeling methodologies. MINI-TUTORIAL ON NON-PARAMETRIC TESTS The Empirical Distribution Function (EDF) and related non-parametric tests can provide “system-wide” information about the significance of individual inputs in multi-layered neural network models. The EDF is defined as follows: n F(x) = 1/n Σ (xj <= x) j=1 where n is the size of the sample, xj, j = 1, 2, .... ,n are the observations, and x is the value being evaluated. For example, if sample X is this set of 12 observations: X = {1, 1, 2, 3, 4, 5, 5, 6, 7, 8, 9, 10}, F(1) = 1/12 * (2) = 0.16667, F(5) = 1/12 * (7) = 0.583333, and F(10) = 1/12 * (12) = 1.0. Tests based on EDF’s are based on the premise that if two samples are from the same general population, their EDF’s will be approximately equal regardless of differences in size. The null hypothesis is of the form: H0: xj xj ± X vk vk ± V, xj # vk, F(xj) ≅ F(vk). The tests require the pooling and ranking of the samples and then testing each sample’s EDF with the pooled sample’s EDF. Suppose V is another sample of 9 observations: V = {1,2,3,3,3,4,4,5,5}. To test if X and V came from the same population, X and V are pooled and sorted to form sample S: S = {1, 1, 1, 2, 2, 3, 3, 3, 3, 4, 4, 4, 5, 5, 5, 5, 6, 7, 8, 9, 10}. Immediately, one can observe that FX(5) = 0.583333 and FV(5) = 1.0, and FS(5) = 1/21 * (15) = 0.714286. But how significant are these differences? Intuitively, it may appear that X and V are drawn from the same population because they share many of the same values. Following are some of the many non-parametric and EDF-based statistics that are easily available using the SAS procedure NPAR1WAY. The first task is to rank order all three sets. Ties are each assigned the average rank for their value. Table 1 has all the sets and their ranks.